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This study involved two phases: first when 
classification was based on the calibration sample, and second in a 
cross-validation setting. Computer-generated data were used. Results 
obtained from rules based on probabilities of group membership were 
compared for accuracy when classifying in the discriminant space and 
in the predictor variable spaces. In the first phase accuracy was 
greater in the predictor variable spaces, while the reverse was true 
in the second phase. In general, rules based on probabilities of 
group membership were approximately equally accurate, and more 
accurate than a rule related to a multiple regression analysis. Other 
findings are also discussed. (Author) 
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Abstract 



This study involved two phases: first when classification x^as 
based on the calibration sample, and second in a cross-validation 
setting. Computer generated data were used. Results obtained from 
rules based on probabilities of group membership were compared for 
accuracy when classifying in the discriminant space and in the predic- 
tor variable spaces. In the first ihase accuracy was greater in the 
predictor variable spaces, while the reverse was true in the second 
phase. In geiieral, rules based on probabilities of group membership 
x^ere approximately equally accurate, and more accurate than a rule 
related to a multiple regression analysis. Other findings are also 
discussed. 
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m EMPIRICAL COMPARISON OF THE ACCURACY OF SELECTED 
MULTIVARIATE CLASSIFICATION RULES 



It is sonetir!:;S of intarest to a researcher to classify individuals 
or objects into one of several categories or classes on the basis of a 
set of observed measures. In so doing the researcher oust choose an 
appropriate multivariate classification rule. From the investigator's 
viewpoint, appropriateness ii:ay be detcrained in terns of 1) having a 
sound thoorecical basis, 2) being capable of acconmodating more t->an 
two criterion populations, and 3) having been adapted, or readily adap- 
table, to conputer progranr-.ing . Many such cultivariate classification 
decision rules have been proposed. This study dealt with five selected 
rules of identifying the population to which an individual my be 
assigned . 

In a nulttvariate classification problem we begin with, say, k 
■subsamples (or groups) of individuals, independently drawn froni k well 
defined populations. There are, say, p measures available for each of 
the N individuals that conprise the total sample. The criterion employed 
In classifying an individual on the basis of the p measures depends upon 
the particular decision rule used. For each of the riaes investigated, 
It is neccssar)' to calculate k nuncrical values— one corresponding to 
each of the k groups— for every individual and then to classify according 
to the criterion of the rule employed. 

In this study, two basic assumptions were made: 1) the k populations 
are p-variate norral, and 2) the k population covariance matrices are 
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identical.'' A brie discussion of the rules under scrutiny in this in- 
vestigation follows. 

Rule 1. Assuming equal costs of misclassif ication and making the 
above two assumptions, a (linear) discriminant score of individual n for 
the gth population ij (Rao, 1965, p, ASS) 
[1] log^ III (X^ - Hg)' (X^ - Hg> + l^ge ^g • 

where 

\ = the common population (p x p) covariance matrix, 

= the (p X 1) predictor score vector for individual n, 
V = the (p X 1) centroid for population g., and 
TT = the probability that individual n (selected at random) 
is from population g. 

Since the expression 

would be common to all k scores for a given individual, the equivalent 
of [l], for classification purposes, is 

v\ f^x - ^ p' T"^ p + log ^ . 

Substituting (maximum likelihood) estimates of \ and y yields a dis- 
criminant score of individual n for group g: 

[2] L X* C^"^ X - h X* C^"^ i + log 7T , 

ng -g -n -g ^e g ' 

where 

5^ * (P X 1) vector of observed mean scores for group g, and 



C « Lhe (p X p) pooled \:ithin-eroups covariance matrix based 
on the Ba:::ple of Vn individuals. 

Here the elencnts of the matrix C are given by 



_ i:i 



where 
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^ij nil ^>^ign ^ V ^"jsn- V' 

with . 

= the nunber of individuals in group g, 

^ign " ^^-^^e predictor variable i for individual p of 
group g, and 

^ig ^ ^^^'^ predictor variable i for group g. 

Of course, since lanple estiI^^tes are used, an optiraum solution cannot 
be claimed. 

The first rule Investigated my be stated" as 
[3] : L^g^ L^^, , 8,g' . i, k ; g ^ g' . 

This is a maximbra likelihood method of classification in the sense that 
It is equivalent (when population values are known) to the x;ule which ■ 
assigns the individual with measurements X to that population for which ' 

c 

the posterior probability of population membership has the highest value. 

RulcJ^. Another indirect application of the criterion of highest 
probability is as follo'.,'e (sec Coolcy and Lohncs, 1971, p. 264). For 
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each individual n the values of k quadratic forms are determined; these 
quadratic forms are given by 

ng -n g -n * 

where 

= the (p X p) covariance matrix for group g, and 

If sampling is from a p-variate normal population, the resulting values 
of [h] follow a chi-square distribution with p degrees of freedom (Rao, 
1952, p. 55). Geometrically, the value of [k] is a measure of the close- 
ness of Individual n's profile point. (X^^, X^^) , in the p~space 
of group g to the centroid of group g. Since the tabled probability of 
a given chi-square value is the likelihood of obtaining a larger value, 
this probability may be considered the proportion of individual points 
that would be expected (in the long run) to lie beyond the locus of 
density (or "centour") on which this particular individual's point lies. 
The "centour score" for an individual, then, yields an estimate of the 
percent of individuals in a population that are further from the centroid. 
The higher the group "centour score" of an individual, the greater is 
his similarity to that group. Hence, this decision rule involves assign- 
ing individual n to that population for which his centour score is 
highest, or equivalently , to that population which yields the lowest chi- 
square sample value given by [1*]. Symbolically, this rule may be stated 
as 
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^2 ^'^ng^^g^' g> g' - 1. ...» k ; g ^ g\ 

Heterogeneity of dispersioa eaters into [5] in a somewhat un^ 
desirable way in that the greater the dispersion for a particular sample 
the greater the likelihood that individuals of unknown classification 
will be identified vith it. Nor does [5] take into account prior prob- 
abilities, 7T , of population membership. Only if the sample dispersion 
matrices are identical and the prior probabilities are the same does [4] 
result in a minimum number of misclassifications . One way of adjusting 
for heterogeneity of dispersion and prior probability consists of modify- 
ing [5] (see Tatsuoka, 1971, pp. 222-225 for other modifications of [5]): 

Rule 3. A more direct application of the classification criterion 
of highest posterior probability of population membership is the basis for 
the third decision rule (see Cooley and Lohnes, 1971, p. 267). Here a set 
of hypotheses regarding population membership of individual n is involved. 
One of the hypotheses is to be retained and the others rejected. The fol- 
lowing notation for determining the likelihood of such an hypothesis is 
used: 

I7] P(Hg |X^) , g = 1, k. 

This denotes the probability of hypothesis g, given the score vector of 
individual n. Hypothesis g, H , states that individual n belongs to pop- 
ulation g. For each individual there would be k such hypotheses, and that 



hypothesis for which the likelihood is a maximum is selected. 

The conditional probabilities of [7] can be computed from Bayes' 

formula, provided the prior probabilities are kno\^ or can be estimated. 

Let P(X I H ) denote the conditional probability of observing the score 

vector X^, given that is true; i.e., the probability that individual 

n selected at random from population g will have the particular combina^ 

tion of predictor scores, (technically, within a "small" neighborhood 

of X )• Then the conditional probability that H is true, given that 
--n g 

observation vector X for individual n was obtained is 

-n 

7T P(X I H ) 

[81 (H^ I X„) . , " . 8 . 1. .... k. 

Under the assumption that the predictor variable vecLors follow a p- 
variate normal distribution in each of the k populations, [8] may be 
expressed as 

TT Id I '"^exp('-^ ) ' 
191 P^^ I y . -T^^-i ~^ .g-l. ....K 

where X^^y is defined by [4]. Formula [9] gives the (posterior) prob- 
ability that individual n with the score vector X , selected at random 

-n 

from the entire sample, will be a member of population g. Thus, a third 
classification rule is to assign an individual to that population for 
which his posterior probability of population membership is largest. 



EKLC 



7 

Such a rule may be orpressed as 

[10] : P > P t , g> g' = 1 k ; g g'- 

J ng ng 

Since [9] utilizes information regarding both prior probabilities and 
differences in dispersions, [lO] produces the same results as those ob- 
tained by using [6] (Huberty, 1969, p. 79). 

Since the denoninators of P and P , for a given individual are 

ng ng 

Identical, classification according to Rule 3 may be equivalent ly per- 
formed by assigning individual n to that population g for which the 
value of 

Tr Id f\xp(-h ) 
g' g' ^ ng 

is a maximun. And since the logarithm function is monotonic increasing, 
this is equivalent to traximising 

log 7T - h ios>. Id I X d"'-'" (X - X^). 

^e g e ' g ' ^ -n -g' g -n -g 

If D is replaced by i and X by u , then this is the same as maximzing 
g ^g "g 

[l]. That is, Rule 1 and Rule 3 are the same except for the sample co- 
Variance matrix used in coniputing the k values of each classification 
statistic. 

It is of interest to note that according to Rule 2 and 3, classifi- 
cation is pcrfomod in the p-dii!>cnsional predictor variable -epaces. To 
classify in the reduced or discriminant space, it is necessary to deter- 
mine linear composites of the original scores, i.e., discriminant function 
Values (see Cooloy and Lohncs , 1971, Ch. 9), and to separate the new 



8 

space defined by these functions into k mutually exclusive regions. 
(These function values are not to be confused with the discriminant 
scores mentioned in relation to Rule 1,) To determine the dimension of 
the reduced space either the eigenvalues of the usual determlnantal equation 
may be subjected to a significance test, or a subset of the non-zero 
eigenvalues that accounts for a large percent, say 90, of the discrimina- 
ting power of the predictor variable may be chosen. It was decided, for 
this study, to employ Rartlctt's test of significance (Rao, 1952, p. 373). 
(An a- level of .10 was used.) As Tatsuoka (1971, p. 233) points out, it 
is reasonable to confine attention to only those discriminant functions 
that are statistically significant so as to decrease' the reliance on 
apparent differences among the criterion groups due to sampling error. 

Rule 4 . Tlie formulation of the classif icatory problem as conceived 
by Knutsen (1955) and Horst (195oa) involves finding separate regression 
equations contrasting each criterion group in turn with all others. In 
finding the regression equation corresponding to group g, the dichotomous 
criterion variable assumes the value 1 for individuals in group g and 0 
otherwise. To determine the weights used in Horst's "least squares" 
multiple classification method, the standard procedure is followed. The 
least squares estimates of one set of population regression coefficients 
are given by 

In] . b = t""^ y , 

where 

b = the (p X 1) column vector of sample weights, 



T « the (p X p) total sample deviation score cross- 
products natriXj, and 

V « the (p X 1) colurrjT vector of deviation score cross- 
products of tlie predictors and the (dichotomous) cri-- 
terion, y ~ the deviations being taken from the grand 
nean. 

Here, the Glenents of T are given by 

^ \ 

t,. « Z Z"^ (X. - X.)(X. - XJ, ij = 1, p, 
g=l n=l ^^"^ 3. 38n 3 

where 

= the mean of the scores on predictor variable i for all 
N individuals . 

The jth element of v for group g coat ras ted with the remaining groups is 
given by 

N Ng N N 

To generalize [ll] to represent the k sets of weights, we write 
the (p X k) matrix, B, vzhere 

[12] B - T^^V. 

The gth column of B is the set of weights corresponding to the gth group, 
and the (p x k) matrix V is a similar extension of v. The weights obtained 
from [12] are appropriate for use with deviation scores. If individuals 
are to be classified into one of several populations using raw score data, 
a correction term must be applied in order to achieve group-to-group compar- 
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...se of raw scores instead of deviation scoires does not change 
the "slope" of the hyperplane but only the point of origin. Therefore, the 
regression weights would remain the same. The raw score formula for 
predicting membership of individual n in population g from the score vector 

-n In* pn 

is 

[13] - Y = b' X + c , 

ng -g -n g * 

where 



b = the eth column of B, and 

c = y - b' X. 
g g -g - 



In the latter expression, 

Y = the mean of the criterion measure which, when the gth 

g 

group is the one considered, is simply , and 

X « the (p X 1) vector of means of the p predictor variables 
for the total sample. 

Classification according to this decision rule requires that for 
each individual, k composite (regression) scores be obtained (using 
equation [13]) which are his k predicted criterion values. Since an 
individual's actual criterion score is 1 or 0, depending upon his group 
membership, it follows that an individual n may logically be identified 
with that population for which his composite weighted score is nearest 
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to unity. Tlius , we have 



[lA] 



: I Y^^ - 1.0 I ^ I Y^^, - 1.0 I, g, « 1, k; g ^ g' 



ng 



Rule 5 * The last rule investigated is based upon posterior probability 

of population membership, as in Rule 3. However, the methods of calculating 

the posterior probabilities for the two rules differ. The formula used to 

compute the posterior probabilities involved in Rule 5 is based on '*Case E. 

i = i but unkno\>m, y unkno\m," presented by Geisser (1966, p. 155). (See 
g -g 

also, Cooley and Lohnes , 1971, p. 269.) Under the assumption of p-variate - 
normality, Geisser obtains the "predictive density of a future observation 
(vector) given the available data" via a Bayesian approach. Geisser 's density 
function is 

N 



[15] 



h(X H ) 
-n g 



N + 1 

g 



2 



1 + 



N (x' C^-^ X ) 
g -n 



(N + 1)(N - k) 

8 



- (N - k + 1) 

2 



where all symbols have been previously defined. Thus, the probability 
that an individual n belongs to population g, given that he has a score 
vector X is by Bayes' formula. 



[16] 



Q (H I X ) - — r~ 
^ng ^ g ' -n' k 



TT h(X |h ) 



, g = 1, k. 



Z TT , h (X |h ,) 



Hence, the last rule used in this study may be stated as 



[17] 
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Of course, since for a given individual, the k values of the denominator 

of [16] are identical, individual n may be assigned to that population 

■H-responding to the larg-st value of tt h (X I H ) . 

g -n' g' 

All five of these decision rules were judged to be appropriate 
according to the three criteria mentioned at the outset. 

Procedure 

To effect the simulation of drawing random samples of size N from 
k p-variate normal populations with a known common covariance matrix a 
highspeed electronic computer—IBM System 360, Model 65— was used. In 
this study the number of predictor variables considered was p = 10, and 
the numbers of criterion groups were k = 3 and k 5. 

To obtain the population covariance matrix, ^, and the population 
mean matrix, M^^^, we proceeded as follows. The classical factor analysis 
model (Harman, 1967, p. 15) may be represented by 

m 

Z 

h=l 



m 

^'^^ \l, ^jh U., J = 1, .... 10. 



where z , F^, and U are standardized normal variates, m < 10 and a 

jh 

and dj are real-valued constants. Further assumptions underlying this 
model are 

Gov (F^, F^,) = Gov (F^, U^) = Cov (Uj , , ) = 0, 
where these covariances refer to population values. Hence, 

tl9] 1 = o2 = E (zj) = Z a^^ + d^ . j = 1. ... 10 

J J h=l J ' 
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and 

m 

[20] ^jj' " ^ ^^j ""y^ ^ ^yh ' j> j' = 1* 10; j ¥ 



Thus, I (10 X 10) is obtained using the relationship 



[21] 1 = A A' + , 

^ pop pop pop 



where 



A = a (10 X m) matrix of elements a., , and 
pop . jh* 

D = a (10 X 10) diagonal matrix with diagonal elements d . , 
pop j 



Equation [21] is the matrix expression for equations [19] and [20]. 

m 2 

The coram una lit y ( Z a ) of each of the predictor variables was 
h=l ^ 

arbitrarily set at .75, thus making the reliability of variable at 
3 

least .75. This condition gives a D matrix with all diagonal elements 

pop 

equal to .50. Any A matrix V7hich is consistent with equations [19] 

pop 

^ ^ m 2 

and [20], under the constraint that T. a., = .75, will suffice. 

t^=l 

Separation between the k populations was accomplished by prescribing 
a (10 x k) population weight matrix: 
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W 



pop 



"11 


"12 


"13 


1 "14 


"15 


"21 


"22 


"23 


1 "24 


"25 


"31 


"32 


0 


' "34 


"35 


"41 


0 


"43 


! "44 


0 


0 




J J 


1 0 


55 


61 


0 


0 


1 "64 


"65 


0 


"72 


0 


; "74 


0 


0 


0 


"83 


' 0 


"85 


0 


0 


0 


1 0 


0 


0 


0 


0 


' 0 


0 



From W a (10 x k) population mean matrix, M . was then obtained 

v^v pop 

(see Footnote A) by the relationship 



pop ^ pop 

The non-zero weights were chosen so that the population values, 

gg 

of Mahalanobis' generalized distance between (the centroids of) two 
populations g and g\ were **signif icant"; i.e., the significance test 

presented by Rao (1965, p. A80) would find the sample counterpart, , 

5 

significant for the smallest value of N used in this study. These 
population distances are given by 
[22] A^^. -H,.)' r'(H, -H,.). 
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Corresponding to each total sample size, sample score matrices of 
size (10 X Ng) were generated from each of the k p variate normal popula- 
tions having the common covariance matrix, j). To generate these sample 
score matrices a procedure similar to that suggested by Kaiser and 
Dickman (1962) was employed. A number was selected from a uniform (0,1) 
distribution using a subroutine called RANDU, corresponding to which a 
number from a normal (0,1) "continuous" distribution was located. This 
technique was used to produce the elements of both an (m x N ) matrix F 
and a (10 x N ) matrix U. The subsample score matrix corresponding to 
group g was then obtained (see equation [l8]) using 

.X ^ AF + DU + M , 
g g 

where 

X = the (10 X N ) matrix of "observed" scores, and 

M = the (10 X N ) matrix, the ith row of which contains 

the (constant) value . 

ig 

Thus, in essence, random samples were selected on the orthogonal F and U 
matrices, and the observed scores were obtained by :ba abov*^ transforma-^ 
tion. 

There were two phases in the present study. In addition to the two 
assumptions mentioned at the outset. Phase I of the study was carried out 
under the following three restrictions: 3) the <i priori probabiiities 
of group membership, tt , g = 1, ...k, are identical; 4) the number of 
individuals drawn from each population is the same; and 5) eai:h sample of 
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individuals Is classified on the basis of data for that sample. For both 

k-values the total sample sizes considered were N « 90, 150, 300 and 450. 

Because of the fourth restriction in this phase a random ^^ainpie of sir.^- 

was composed of k subsamples of size ii/n cuctu xa this phase sampling 

from each of the k populations was repeated 100 times for each value of 

N. Restriction 3) lead us to the "uniform ignorance" assumption of tt « 1/k 

g ' 

g 1, k (see Tatsuoka, 1971, pp. 225-226); hence the 7r-values called 

for in computing the probabilities in Rule 3 were deleted. Rule 5 was nat 
included in Phase I because under conditions 3). and 4) it yields results 
that are identical to those of Rule 1 (liuberty, 1971). 

The purpose of Phase II was to empirically investigate the accuracy 
of the five selected classification rules in a situation where cun -lit ions 
3), 4), and 5) were either removed or modified in such a way as to make 
them more compatible with the "real world." In practical applications of 
multivariate classification theory, the number of individuals in each of 
the subsamples thdL .:^--nGnt the k populations of interest often is not 
the same. Hence, the k prior probabilities, estimates of which are usually 
based on the subsample sizes, can not be taken to be identical. Further, the 
investigator often desires to classify individuals whose -proper classifica- 
tion has not been determined at the time when sampling to . place. Thus, he 
seeks a relatively accurate clas£jif ication rule to employ with individuals 
other than those in the "calibration sample." The proportion of misclassif i- 
cations obtained by using the calibration or norming sample— the "apparent" 
error rate—tends to underestimate the "actual" error rate—the long run 
frequency of misclassif ications using replication samples (Geisser, 1970) • 
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Intuitively this seems reasonable since rlas s t ficat i * ^ndl*. ' dua 

in the normtnR naii^^.u n is bascid on a rulti which is not only ia some 

sense optimal for these particular individuals but Is 1 Ilcaly t:o caplml i^e 
favorably on chance sampling fluctuations. It is for these reasonH^ rJjat 
dSAxQ situation of Phase I was altered by the removal or inodif icatiotr^ 
conditions 3), 4), and 5). 

In Phase II of the study the total sample sizes considered were ^' 150, 
300> A50> and 600 when three criterion groups were icwolved and N ^ 3£!0, 
4J50, 60(J when k « 5. A random sample of size N was campiosed of k sa2k^samples 
erf size N^, .o.^ Nj^. For k = 3, the ratios of subsarmple sizes were anibitrarily 
S€t at 9:5:1. For k * 5, the ratios selected! were 12:9:5:3:1. Hlhu: 
alignment of suibsample size to each respective group was made by essipILoying 
a table of xandoim numbers. For example, with k = 5 and N ^ 450, tte assign- 
ment was: =^ 75, = 15, = 45, ^ 180, and = 135. 

Rather than apply the selected rules to the sampd:e of ^'individaials" on 
\^ich the -classification statistics are determined, the nGcessar>^ nmEtrix 
calculations were performed on the first sample and classif icationr^according 
tct^each of the five^ selected rules was carried out on the succeedfesag lOvO 
samples. Thus the comparative accuracy of the rules was determined! in a 
"cxoss-validation*' setting. 

Data Analysis 

The criteriion used to judge the accuracy of each of the five q^eciision 
rules is the number of correct classifications over k groups. In ana^rzing 
the accuracy of population identification provided by the rules, a diei»crip- 
tive analysis wa^. performed on the number of correct classifications <mez: 
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all k groups for each value of N and each value of k. Frequency distri- 
butions of the number of correct classifications for each run of the 
experiment were obtained and comparisons were made of 1) the means and 
standard deviations of these distributions, and 2) the intercorrelations 
among the numbers of correct classifications obtained from each of the 
rules investigated. 

An analysis. of variance was employed in comparing the results of the 

rules statistically. The ''treatr.iGnts-by-subjects" design described by 

8 

Lindquist (1953, Ch. 6) was applied to the numbers of correct classifi- 
cations to compare the results of Rules 2 and 3 in both the predictor 
variable spaces and the discriminant space. The "treatments'* were the 
rules being investigated, while the "subjects" were the (100) runs, llie 
following five hypotheses were tested; 

"o = ^2V = ^3V " ^3D ' 
"O = '^2V = ^2D ' 

where 

^IV ~ '■^^ population mean of the numbers of correct classifi- 
cations using Rule i (i = 2 or 3) in the predictor variable 
spaces. 



(i) 

(ii) 

(iii) 

(iv) 

(v) 



^2 

^iD ^ ^'^^ population mean of the numbers ct .:.^rrect classlti 
using Rule i (i = 2 or 3) in the dis.rlrinant space, 

^i? ^^^^ population mean of the numbers ot corre:t clai : tf .1 cat: m.; 
using Rule i (i = 2 or 3) in the spatx* deterniiiioc. l y u c .c 
suits of the tests of hypotheses (ii) and (lii), '3ivi 

^2 5^ population ir.ean of the numbers of correct classifications 
for the rule and space that is expected to be *'bcst" in 
accord with the results of the test of hypothesis (iv). 

Hypotheses (ii) and (iii) were tested to determiriC ? r c'-^ -:':7"^.ntr£'7'ir'-. -i-iowi-r 
is different vhen classifying in the discriminant Spaco instead c i! in tb: 
predictor variable spaces. Of course, the test of hypothesi-:? (iv) \<a- 
only made if the first hypothesis was rejected. The pre jeourc ij\ erl tu 
test hypotheses (ii) , (iii), and (iv) is that of Schcff^^ (1939, v • 
(As mentioned previously Rule 5 was not considered in p;:asc. 1 of thr s.tfcy,) 

Results 

Many of the reported studies in the behavioral sciences that employ 
"multiple group discriminant analysis'* were done for the purpose of assign- 
ing an individual to one of a finite number of populations to which he 
may belong. This assignment was made on the basis of a set of characteristics 
observed on the individual. Sometimes, especially in personnel work ard 
career planning studies, discriminant analysis is used primarily n; a .u;.r:.: 
of analyzing group and variable 'differences ^ and the classification aspect 
of the analysis is considered as a by^produqt. The primary concern of the 
present study, however, was one of identification; that is, how accurately 
do different decision rules identify an individual as a member of one of 



several criterion populatioas? Of course, one way to ansv;i?r Liil-' cju.-.Lion 
and, hence, compare the efficiency of these rules, is to cUUtirmiiie for each 
rule the proportion of correct classifications of a fixed sample of sub- 
jects. For a given set of rules being investigated the "best" one may be 
defined as that one which consistently, over repeated sampling, produces 
the highest proportion of correct classifications. 

To aid in the interpretation of the results the percents , :i:<ic]Lis, and 
standard deviations based on chance expectations are included in the 
appropriate tables. The resulting values are based on the multinomial 
situation present here. The expected proportion of correct classifications 
for a given N-value across all k groups is 

1 

p = ~ z: TT N . 

S g 

In this study tt was taken to be N /N. Of course, in Phase I, p = 1/k. 
g S 

The expected number of correct classifications is given by Np ; and ihr> 
expected standard deviation of the distribution of correct classl r ications 

is /Np(l-p) . 

The results of this study will be discussed separately for each phase 
and each k'-value. 

Phas e I — k = 3 . The p^>rcents, means, and standard devir!, { rr^q of the 

numbers of correct classifications, as reported in Tablo I, rjfloct the 

Insert Table 1 here 

efficiency of each of the four rules studied* The means ^/v'ea in TriMe 1 
indicate that Rules 2V and 3V (classifying in the predictor variable spaces 



according to Rules 2 and 3, respectively) identify population membership most 
accurately. Itiis result was also evident from a comparis^on of the frequency 
distributions of the numbers of correct classifications for each rule, l^ie 
distributions resulting from classification using Rules 2V and 3V were quite 
similar. The corresponding distributions for Rules 1, 2D, 3D, and 4 v/ere 
also similar though markedly lower. This similarity was also apparent from 
the intercorrelatlons among the numbers of correct classifications according 
to the different rules. Hypothesis (i) wa5^ clearly rejected for all four 
Values of N; the value of each mean square ratio (MSR) was at least 500. 
The differences indicated in hypotheses (ii) and (iii) were both highly 
significant — as judged by the Scheffc procedure — with the use of the pre- 
dictor variable spaces being favored over the discriminant space. Since 
both hypotheses (ii) and (iii) were rejected for all N-values , the results 
of this phase of the study do not agree with those reported by Lohnos (1961). 
That is, a significant loss in discriminating power was observed when 
classifying according to Rules 2 and 3 in the discriminant space rather than 
in the predictor variable spac^^s. 

The outcomes of the tests of hvpothescs (ii) and (ill) indicate that 
hypothesis (iv) could he stated as 

(iv) Hq : ii^^, - Vi3^, . 

In no case could this hypothesis be rejected. However, because in each case 
the observed mean of Rule 3V was greater than that of Rule 2V, the fifth 
hypothesis was stated as 

(v) Hq : = 1.3^, = li^ . 



The values of the four MSRa in the testing of (v) were in no case less than 
600. Hence, in each case hypothesis (v) was rejected; and Scheff^'s method 
was again employed in making follow-up pairwise comparisons, Differences 
between Rule 3V means on the one hand and Rule 1 and Rule 4 means on the 
other vera highly si^r.if icariC ir.dicacing that Rule iV couid be expaccci 
to produce (in the long run) the highest number of correct classifications. 
Further, it was found that the classification accuracies of Rules 1 and 4 
were not significantly different. This result is not consistent with that 
found by Knutsen (1955)^ who concluded, from his single sample, that 
Rule 4 was more accurate than Rule 1. 

Phase I — k = 5. Except for the fact that the numbers of correct 
classifications were considerably larger, the results for five groups were 
very similar to those for the three-group case. As seen in Table 2 the 
reflected trends (from rule to rule and along N-values) are parallel to 
those results when k = 3. The outcomes of the tests of the five hypotheses 
of interest were also the same for the five-group situation as for the case 
of the three groups. 

Insert Table 2 here 



Phase II — k - 3 > The percents, means and standard deviations of the 
numbers of correct classifications are given in Table 3. These results 
indicate the superiority of Rules 1, 3D, and 5 in terms of accuracy of 
identification of population membership. That these three rules tend to 
produce similar degrees of classification accuracy is apparent from tables 
(not presented here) of the intercorrelations and frequency distributions 



of the numbers of correct classification. The correlations among these 
rules ranged from .85 to .99. It was pointed out earlier that in a 
"true*' sense Rules 1 and 3 are equivalent. That is, the classification 
statistics used are the same except for the sample covariance matrix used 
in computing the k values for each individual, Furthcirmore , it has been 
showT\ that when the N -values are the same, Rules 1 and 5 are equivalent. 
Thus, descrepancies among the results yielded by Rules 1, 3, and 5 are 
merely a function of the "goodness" of the sample estimates of ^ and of 
rr , g = 1' , . . . , k . 

Based on the analysis of variance results (no MSR was less than 200) 
hypothesis (i) can not be considered tenable in any of the four sample 
size cases. As judged by Schcffe's procedure, the differences indicated 
by hypotheses (ii) and (iii) were highly significant in all cases. Rule 2 



Insert Table 3 here 



in the predictor variable spaces produced a greater number of correct 
classifications than in the reduced or discriminant space for all N-values* 
Rule 3, on the other hand, appears to be more accurate for all N-values 
considered in this study if it is applied in the reduced space. This latter 
result is not consistent with that found in Phase I. 

The outcomes of the tests of hypotheses (ii) and (iii) indicate that 
hypothesis (iv) can be stated as 

(iv) Hq : y3jj = ' 
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This hypothesis was rejected in all four situations, with Rule 3 (in the 
discriminant space) being favored for each sample size. This result lead 
to the following statement for hypothesis (v) : 

(v) Hq : = = = .,5 . 

Since the >lSRs resulting from the analysis of variance in the testing of 
this hypothesis were at least 700 for any N-value, these four rules were 
not judged to be equally accurate. The method of Scheffe was again 
employed in making the follow-up pairwise comparisons. Differences 
between Rule 4 means on the one hand and means of Rules 1, 3D, and 5 
on the other were highly significant. In all situations the significant 
results indicated that Rule 4 would be expected to produce (in the long 
run) fewer correct classifications. Further, it was found that the 
classification accuracies of Rules 1 and 3D, 1 and 5, and 3D and 5 were, 
in each case, not significantly different. The results of the tests 
following hypothesis (v) are summarized in Table 4. 



Insert Table 4 here 



EKLC 



Phase II — k = 5 * As in the three-group case. Rules 1, 3D, and 5 
produced the greatest number of correct classifications (see Table 5). 
Again hypothesis (i) was rejected in each case (all MSRs were at least 
300), and of the four rules considered in this hypothesis results of 
applying Scheffe' s procedure indicated Rule 3D as being the most accurate. 
Table 6 summarizes the results of the application of Scheffe's individual 
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comparison test following significant MSRs (all ratios were at least 
1000) of the tests of hypothesis (v) . Note that for N = 450 Rules 1 
and 5 ai- more accurate than Rule 3 (in the discriminant space). For 
the other two N-values , the comparisons of Rule 1 versus Rule 3D and 
Rule 5 versus Rule 3D indicated no significant differences in classifi- 
cation accuracy; and Rules 1 and 5 were equally accurate for all N-values. 
Again, as in the case of k = 3, Rule 4 was found to be less accurate 
than Rules 1, 3D, and 5. 

Insert Tables 5 & 6 here 



It should be noted that sampling performed with different orderings 
of subsample sizes produced very similar results; that is , nearly the 
same rankings of the rules (with respect to classification accuracy) 
resulted. Also, it is of interest to note that the results of applying 
Rule 5 in the reduced space were almost identical to its application in 
the predictor variable spaces; the mean of the seven differences of mean 
number of correct classifications over both valuer of k (four for k = 3, 
three for k = 5) was .79 and the mean correlation between classification 
accuracy in the different spaces over both values of k was .95. 



Discussion 



Any remarks concerning the results of Phase I must necessarily be 
made in light of a situation characterized by the following restrictions: 
1) the k criterion populations are p-variate normal, 2) the k population 
covariance matrices are identical, 3) the a priori probabilities of popu- 
lation membership are identical, 4) the number of individuals drawn from 
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each population is the same, and 5) each sample of individuals is 
classified on the basis of the sample data. It was judged that the 
populatior covariance (i.e., correlation) matrix used in this study 
was not atypical of those found in practice (Cochran, 196A, p. 186). 

Perhaps one of the most striking findings of Phase I was the 
decreasing accuracy of classification with increasing N. One possible 
reason for this is the potential bias that may have been brought into 
the situation because of the use of each observed group covariance 
matrix rather than the pooled matrix and/or the sampling process itself. 
If a bias was present the larger sample sizes would reflect this more 
distinctly causing lesser accuracy. Another explanation which accounts 
in part for the outcome at hand is as follows. Let us consider Rule 2V 
in which the classification of individual n is based on the distance of 
his profile point to the centroid of each of the k groups. In the case 
of group g, this distance is equal to the mean distance between n's pro- 
file point and each of the profile points for the N individuals in the 
group. ^ In the event that individual n is a member of group g, one of 
the distances entering into the mean necessarily has a null value. The 
effect of this would be expected to be inversely related to the size of 

N . In a two-group case the *'true" proportion of correct 
g 
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classifications is easily estimable, and in the situation of this 
study is obtainable directly using C(A/2) where « is the standard 
normal distribution function and defined by L222 (Dunn, 1971). 

The simulation procedure was repeated exactly as in Phase I except 
for the consideration of only two criterion populations (k = 2). The 
proportion of correct classifications also decreased with increasing 
total sample size, and the proportion seemingly approached ^)(A/2), 
which was .755 in this case. As pointed out by Click (1972), computing 
the probability of correct classification for k > 2 involves evaluating 
an integral over a p-dimensional region, the form of which may vary 
with each rule. 

That, for the given situation, all of the rules yielded results 
that were better than if the classifications were made at random is 
clear from a comparison with the chance expectations, A statistical 
test of "discriminatory power" is available (Press, 1972, pp. 381-383) 
but, because of the obvious outcomes, is not reported here. 
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The fact that the numbers of correct classifications vere larger for 

k 5 than for k = 3 does not necessarily imply that classification 

accuracy for comparable N-values improves as k increases* One reason 

for this result in this phase of the study may be that the separation 

of the populr>cions (as measured by Mahalanobis' generali2ed distance 

function, A^) in the k = 5 case was greater than in the case of k - 3, 

For k « 3 the mean tp- was 1.55; for k = 5 the mean was 3,86, A closer 

examination of Tables 1 and 2 reveals that under the conditions of the 

present study the accuracy of classification from use of most of the 

rules appears to depend upon N rather than k* For example, the applica- 

tion of Rule 2 in the variable spaces yielded 81,1% accuracy for k = 3 

and N = 30 versus 80.1% for k = 5 and N =30; 74.0% for k = 3 and 
8 g 

N 50 versus 71.1% for k = 5 and N =60; 67.7% for k = 3 and N = 100 
g g g 

versus 67.8% for k = 5 and N = 90. Such a conclusion would be valid 

g 

if the distances betx^een the population centroids vere the same for both 
k = 3 and k - 5, but they x^erc not. 

The final remarks pertaining to the results of Phase I have to do 
with classification in the discriminant space as compared to classification 
based directly upon the original p variables^ It is a fact that the 
results of classification based on Rules 2 and 3 (assuming identical 
population covariance matrices) arc the same regardless of whether the 
original p variables or the k-1 discriminant functions are- used in compu- 
ting the chi-square values.' Tlifs is true because the differences among 
the group centroids are e>:haust(jd by Che k-1 discriminant functions; thus 
the distance between an individual and a group centroid would be the same 
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In the p-dinensional spaces of the total sample as in the discriminant 
space. So, first, differences in classification accuracy ©ay be expected 
because of sampling fluctuations. It is important to note that with 
the use of each group covariance matrix, rather than the pooled covarlance 
matrix, such fluctuations are indeed taken into consideration. Secondly, 
since some of the discriminant functions were considered "nonsignificant" 
the dimensionality of the discriminant space was sometimes less than k-1, 
particularly in the five-group case. 

In Phase II of the study restrictions 3), 4), and 5) mentioned at 
the outset of this section were removed. The marked weakness of Rule 2D 
compared to Rule 3D in Phase 11 deserves some comment. The same chi-square 
values^ based on the sane number of discriminant functions, were operated 
on by both of these rules. Upon examining the results according to sub- 
sample size it was found that Rule 2D was as accurate or slightly more so 
than Rule 3D for smaller N -values, whereas Rule 3D yielded considerably 

better accuracy for larger N ^values. That is, by taking into consideration 

5 

the prior probabilities, as does Rule 3D, it was often found that the 
largest of the posterior probabilities was associated with the largest 
subsample size. That this may result is suggested by Tatsuoka (1971, 
p. 226). 

A result' of Phase II which differed from that of Phase I involves the 
accuracy of Rule 3 when classifying in the discriminant space versus 
classifying in the predictor variable spaces. In Phae^o II of the 
study classifying in the discriminant space Vi^l^*^^ better results* It 
was conjectured that this superiority of accuracy is a matter of the 



3a- 

stability of the discriminant functions (over repeated sataplitig). By 

"stability" is meant that since a discriminant function is a linear 

composite of the predictor variables it would exhibit less sanpling 

fluctuation over repeated sanpling than the single variables wliich enter 

into the conposite. However, an inportant consideration here is that the 

classification statistics used in Rule 3V are based only on the N individuals 

g 

in group g (to deterziine D ) vhereas the statistics used in Rule 3D are 
based on the total sarn^le of individuals (to detcraine the discrininant 
functions). 

To surziarize, then, in terms of classification accuracy It can be 
expected that, under the conditions of either Phase I or Phase H, Rule 2 
applied in the discriminant space and Rule A will not in the long run do 
as well as the other rules. Since a true classification problem is one 
which involves assigning as yet "unlabeled" individuals to one of a number 
of well-defined criterion populations it may be well to only consider 
classification accuracy of rules applied in a cross-validation setting. 
(However, a proportion of correct classifications when based on a '^large" 
calibration sanple nay be used as an index of discriminatory power of a 
set of predictors.) In this sense it was found in Phase II that Rule 1, v 
3D, and 5 can be expected to yield the largest, and nearly the sane, 
number of correct classifications. 
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FOOTNOTES 



It is recognized that an alternative multivariate model is 
sometimes considered; when a ''total population*' concept makes sense, 
we are dealing with k *'subpopulat ions . The difference may bear on 
interpretation of results. 

'"This is read as follows: "Assign individual n to population g 
if his discriininanc scorti, L^^, corrosponding to population g is largest, 

3 

Although the underlying statistical model of discriminant 
analysis assumes no errors of measurement, in this study the predictor 
variables are allowed to be fallible in that each variable has a 
maximum error variance of .25. Allowing for non-'zero unreliability 
is in agreement with most studies involving psychometric measurement. 

^The choice of this array of zero and non-zero weights was made 

in view of a study which treats the variable selection problem. The 

A and t matrices, tlie routine employed in arriving at A > and 
pop ^ P^P 

the W and M matrices have been deposited with the National 
pop pop 

Auxiliary Publications Service. Order Document No. from 
National Au)Ciliar>^ Publications Service of the American Society for 
Information Science, c/o CCM Information Sciences, Inc. 22 West 34th 
Street, New York, New York lOOOl. Remit in Advance $ for photo- 

copies or $ for microfiche and make checks payable to: Research 

and Microfilm Publications, Inc. 



A low significance level (a .10) was employed so as to subject 
the classification procedures to a relatively severe test of efficiency. 
If the separation of the populations is great, many classif icatory 
schemes would appf;:ar to be equivalently efficient. Also, it was of 
Interest to keep the probability of making a Type II error-- claiming 
the groups are not significantly different when they actually are — 
low; hence, a large a- value was used. 

■^The smallest total sample size that could be used with k » 3 
was 150 and with k = 5 was 300 since the given ratios require one 
subsample of size 10; with p * 10 a smaller number of '^individuals'' 
would produce a dispersion matrix, D, whose determinant is zero 
(since the rank of D would be less than 10). 



A codification of the least squares technique used in arriving 
at the classification statistic for Rale 4 was required because of the 
unequal subsanple sizes er.plcyed in Pliase II (sco Horst, 1956b). 

g 

The conservative test suggested by Box (195A) was employed in all 
situations. 

mis steins fror: the fact that 

- X - C(X^ - ) + (-\ - X, ) + ... + (X - \, )2. 

-n -g -a -Ig -n -2g g 



TABLE 1 



Phase I 

Percents, Means, Standard Deviations of the Number of Correct 



Classifications tor k = J 



Rule 



2V 2D 3V 3D 



Chance 
Expectations 



66.6% 



81.1% 66.0% 83.0% 66.9% 66.2% 33.3% 

90 59'.96 73.04 59.44 74.70 60.19 59.60 30 

(4.44) (3.57) (5.31) (3,54) (4.93) (4.46) (4.47) 

63.4% 74.0% 65.5% 75.0% 63.9% 63.3% 33.3% 

150 95.06 110.93 95.19 112.46 95.79 94.94 50 

(5.87) (6.28) (5.48) (5.92) (6.17) (5.73) (5.77) 

61.7% 67.7% 61.9% 68.4% 61.8% 61.6% 33.3% 

300 185.16 203.21 1S3.66 205.34 185.31 184.90 100 

(7.74) (8.69) (7.86) (9.02) (7.67) (7.50) (8.16) 

61.2% 65.2% 61.2% 65.8% 61.4% 61,2% 33.3% 

450 275.46 293.59 275.35 296.31 276.26 275.40 150 

(10.03) (11.11) (10.12) (10.59) (10.16) (10.09) (10.00) 



^Standard deviations are given in parentheses. 



TABLE 2 
Phase I 

Percents, Means, and Standard Deviations of the Number of Correct 

Classifications for k = 5^ 



Rule 







1 


2V 


2D 


3V 


3D 


4 


Chance 
Expectations 


N = 


90 


69.1% 
62.20 
(4.53) 


89.4% 
60.50 
(3.25) 


71.0% 
63.93 
(5.22) 


91.0% 
81.86 
(2.93) 


72.4% 
-65.16 
(4.79) 


68.4% 
61.53 
(4.50) 


20.0% 
18 

(3.79) 


N = 


150 


66.9% 
100.38 
(5.28) 


80.1% 

12C-iS 
(4.73) 


68.5% 
102.69 
(5.68) 


81.2% 
121.87 
(4.40) 


68.9% 
103.41 
(5.18) 


66.1% 
99.19 
(5.12) 


20.0% 
30 

(4.90) 


N = 


300 


63.6% 
190.67 
(8.26) 


71.1% 
213.41 
(7.98) 


64.2% 
192.64 
(8.34) 


71.8% 
215.36 
(7.66) 


64.6% 
193.72 
(8.34) 


63.2% 
189.52 
(8.66) 


20.0% 
60 

(6.00) 


N = 


450 


62 . 8% 
' 282.67 
(10.17) 


67.8% 
304.91 . 
(10.43) 


63.4% 
285.40 
(10.20) 


68.3% 
307.43 
(9.97) 


63.7% 
286.49 
(10.16) 


62.5% 
281.05 
(9.71) 


20.0% 
90 

(8.48) 



Standard Deviations arc given in paronthesos. 



TABLE 3 
Phase II 

Percents, Means, and Standard Deviations of the Number of Correct 

Classifications for k = 3 



Rule 







1 


2V 


2D 


3V 


3D 


4 


5 


Chance 
Expectations 


N = 


150 


66.2% 
99.35 
(5.36) 


62.3% 
93.50 
(5.29) 


57.1% 
85.59 
(6.81) 


63.5% 
95.26 
(4.84) 


65.8% 
98.72 
(5.12) 


56,0% 
84.00 
(6.48) 


66.6% 
99.90 
(4.99) 


47.6% 
71 

(6.12) 


N = 


300 


68.3% 
204.79 
(6.67) 


64.0% 
192.01 
(8.13) 


56.8% 
170.35 
(9.08) 


64.9% 
194.81 
(7.75) 


68.2% 
204.71 
(6.39) 


56.9% 
170.77 
(8.64) 


68.2% 
204 . 58 
(6.51) 


47.6% 
143 
(8.65) 


N = 


450 


69.6% 
313.17 
(9.01) 


63.8% 
287.04 
(9.64) 


58.5% 
263. 38 
(10.85) 


67.8% 
304.93 
(7.45) 


69.6% 
313.11 
(8.72) 


59.7% 
268.83 
(10.26) 


69.6% 
313.16 
(8.85) 


47.6% 
214 

(10.59) 


N » 


600 


69.8% 
418.65 
(10.23) 


64.2% 
385.02 
(11.06) 


62.0% 
372.22 
(11.72) 


67.8% 
406.65 
(10.89) 


69.8% 
418.75 
(10.34) 


59 . 3% 
355.74 
(12.50) 


69.8% 
418.95 

(10.13) 


47.6% 

285 

(12.23) 



^Standard deviations are given in parentheses. 



TABLE 4 

Mean Differeaces rnd Scheffe 
Critical Values for k = 3 













^3D"^4 


S"^3D 




Critical 
Valued 


■N = 


150 


.63 


15.35 


.55 


14.72 


1.18 


15.90 


1.84 


N = 


300 


.08 


34.02 


-.21 


33.94 


-.13 


33.81 


3.05 


N = 


450 


.06 


44.34 


-.01 


44.28 


.05 


44.33 


3.02 


N = 


600 


-.10 


52.91 


.30 


53.01 


.20 


63.21 


3.83 



ERIC 



TABLE 5 
Phase II 

Percents, Means, and Standard Deviations of the Number of Correct 

Classifications for k = 5^ 



Rule 







1 


2V 


2D 


3V 


3D 


4 


5 


Chance 
Expectations 


N = 


300 


65.9% 
19 7 . 60 
(7.65) 


SC.. 4% 
181.26 
(7.16) 


59.0% 
176.93 
(S.73) 


60.8% 
182.33 
(7.05) 


65 . 3% 
195.95 
(7.15) 


54.6% 
163. 77 
(8.38) 


66.0% 
197.88 
(7.45) 


28.8% 

87 

(7.85) 


N ■= 


450 


65.5% 
294.82 
(10.34) 


58.4% 
262.77 
(10.86) 


56.6% 
254.84 
(11.56) 


61.1% 
274.92 
(8.76) 


64 . 8% 
291.44 
(9.92) 


58.4% 
262.90 
(10.84) 


65.6% 
295.05 
(10.13) 


28.8% 
130 
(9.61) 


N = 


600 


65.0% 
390.04 
(11.51) 


58.6% 
351.5 7 
(13. 38) 


59.9% 
359.34 
(11.78) 


61.8% 
370.80 
(13.07) 


64.6% 
387.34 
(11.47) 


54.5% 
326.83 
(11.68) 


65 . 0% 
389 .99 
(11.530) 


28.8% 

173 

(11.10) 




standard deviations are given in parentheses. 



TABLE 6 

Mean Differences and Scheffe' 
Critical Values for k = 5 











^5-^1 


^3D~^4 


^-^3D 




Critical 
Valued 


N = 


300 


1.65 


33.83 


.28 


32.18 


1.93 


34.11 


2.63 


N = 


450 


3.38 


31.92 


.23 


28.54 


3.61 


32.15 


3.14 


N = 


600 


2.70 


63.21 


-.05 


60.51 


2.65 


63.16 


3.78 
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